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Abstract 

As  the  United  States  Department  of  Defense  continues  to  increase  the  number  of 
Remotely  Piloted  Aircraft  (RPA)  operations  overseas,  improved  Human  Systems  In¬ 
tegration  becomes  increasingly  important.  RPA  systems  rely  heavily  on  distributed 
team  communications  determined  by  systems  architecture.  Two  studies  examine  the 
effects  of  systems  architecture  on  operator  workload  of  US  Air  Force  MQ-1/9  oper¬ 
ators.  The  first  study  ascertains  the  effects  of  communication  modality  changes  on 
mental  workload  using  the  Improved  Research  Integration  Pro  (IMPRINT)  software 
tool  to  estimate  pilot  workload.  This  study  shows  that,  through  the  proper  allocation 
of  communication  between  modalities,  workload  can  be  reduced.  The  second  study 
uses  IMPRINT  to  model  Mission  Intelligence  Controllers  (MICs)  and  the  effect  of 
the  system  architecture  upon  them.  Four  system  configurations  were  simulated  for 
four  mission  activity  levels.  Mental  workload,  monitoring  time  and  the  number  of 
delayed  tasks  were  estimated  to  determine  the  effect  of  changing  system  architecture 
parameters.  Literature  and  MIC  interviews  provided  parameters  for  the  model.  The 
analysis  demonstrates  that  the  proposed  changes  have  significant  effects  on  workload 
and  system  monitoring  time. 
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DISCRETE  EVENT  SIMULATION  OF  DISTRIBUTED  TEAM 

COMMUNICATION 

I.  Introduction 

The  U.S.  Department  of  Defense  (DoD)  has  been  a  major  developer  of  RPA  tech¬ 
nology  which  it  has  used  primarily  for  intelligence  operations.  The  DoD  continues 
to  fund  RPA  development,  even  in  an  otherwise  austere  acquisitions  environment 
(Defense,  2011).  Congress  has  passed  legislation  to  begin  integration  of  civil  and 
commercial  RPAs  into  the  national  airspace  system  (House.,  2011). 

This  research  makes  a  case  for  including  communication  requirements  in  future 
system  designs  through  two  studies  which  examine  the  effects  of  architecture  changes 
on  communication  related  workload.  The  Improved  Research  Integration  (IMPRINT) 
software  tool  was  used  to  create  executable  communications  architecture  models  in 
both  studies.  The  models  draw  from  the  experiences  of  qualified  MQ-1  Predator  and 
MQ-9  Reaper  operators  interviewed  by  the  author,  and  represent  the  systems  they 
use. 

MQ-1  and  MQ-9  operators  currently  use  military  internet  relay  chat  (mIRC), 
radio  and  intercom  systems  to  communicate  with  co-located  and  distributed  teams. 
Modalities  are  fixed  in  current  operations,  meaning  that  vocal  messages  are  always 
heard  and  visual  messages  are  always  seen.  The  first  study  is  centered  around  the  MQ 
pilot  and  explores  the  ramifications  of  being  able  to  shift  message  allocation  between 
the  auditory-verbal  and  visual-verbal  channels.  It  is  assumed  that  communication 
arriving  on  one  channel  could  be  reallocated  to  another  channel.  Reallocation  could 
presumably  occur  as  a  result  of  automation  or  policy  changes  to  the  way  users  employ 
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communication  channels  within  the  network.  This  allocation  change  shifts  the  mental 
resource  demand,  which  changes  the  workload  induced  by  the  system. 

The  second  study  examines  the  effects  of  architecture  changes  on  workload  for 
the  mission  intelligence  coordinator  (MIC).  Changing  the  number  of  communication 
nodes  and  exclusion  of  secondary  navigation  tasks  constitute  potential  system  design 
changes.  The  current  systems  and  design  changes  are  modeled  to  determine  the  effects 
of  the  changes. 

Effective  and  efficient  communication  is  foundational  to  military  strength  and  is 
especially  important  in  mission  areas  where  many  of  the  collaborators  are  distributed 
around  the  globe  yet  must  communicate  in  real  time  to  complete  the  mission  objec¬ 
tives.  Future  system  requirements  will  need  to  reflect  cognizance  of  how  communica¬ 
tions  overhead  and  team  structure  can  influence  workload  and  mission  performance. 
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II.  Communication  Modality  Allocation 


This  paper  was  presented  at  the  Conference  of  Systems  Engineering  Research 
(CSER)  in  St.  Louis,  Missouri  on  March  23,  2012. 

Allocation  of  Communications  to  Reduce  Mental  Workload 

Travis  Pond*,  Brandon  Webster,  John  Machuca,  John  Colombi,  Michael  Miller, 

Randall  Gibb 

Air  Force  Institute  of  Technology,  Wright-Patterson  Air  Force  Base,  Ohio,  45433, 
USA 

Abstract  As  the  United  States  Department  of  Defense  (DOD)  continues  to  in¬ 
crease  the  number  of  Remotely  Piloted  Aircraft  (RPA)  operations  overseas,  improved 
Human  Systems  Integration  (HSI)  becomes  increasingly  important.  Manpower  limi¬ 
tations  have  motivated  the  investigation  of  Multiple  Aircraft  Control  (MAC)  configu¬ 
rations  where  a  single  pilot  controls  multiple  RPAs  simultaneously.  Previous  research 
has  indicated  that  frequent,  unpredictable,  and  oftentimes  overwhelming,  volumes  of 
communication  events  can  produce  unmanageable  levels  of  system  induced  workload 
for  MAC  pilots.  Existing  human  computer  interface  design  includes  both  visual  infor¬ 
mation  with  typed  responses,  which  conflict  with  numerous  other  visual  tasks  the  pilot 
performs,  and  auditory  information  that  is  provided  through  multiple  audio  devices 
with  speech  response.  This  paper  extends  previous  discrete  event  workload  models  of 
pilot  activities  flying  multiple  aircraft.  Specifically,  we  examine  statically  reallocat¬ 
ing  communication  modality  with  the  goal  to  reduce,  and  minimize,  the  overall  pilot 
cognitive  workload.  The  analysis  investigates  the  impact  of  various  communication 
reallocations  on  predicted  pilot  workload,  measured  by  the  percent  of  time  workload 
is  over  a  saturation  threshold. 
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2.1  Introduction 

Over  the  past  several  decades,  the  US  Air  Force  has  harnessed  and  exploited 
the  immense  tactical  power  that  middle  and  high-altitude  Remotely  Piloted  Aircraft 
(RPAs)  bring  to  the  battlefield.  As  a  consequence,  the  demand  for  RPA  operational 
support  continues  to  increase.  It  is  important  to  realize  that  RPAs  are  part  of  a  com¬ 
plex  system.  The  system  has  many  components  including  one  or  more  air  vehicles, 
ground  control  stations  (GCS)  for  both  primary  mission  control  and  takeoff/landing, 
a  suite  of  communications  (including  intercom,  chat,  radios,  phones,  a  satellite  link, 
etc),  support  equipment,  and  operations  and  maintenance  crews  (USAF  Air  Combat 
Command,  2010).  It  goes  without  saying  that  the  assets  and  requisite  resources  to 
support  those  operations  are  far  from  unlimited  and  personnel  resources,  particularly 
RPA  pilots,  often  prove  a  nontrivial  constraint.  This  inevitably  leads  innovators  to 
seek  out  RPA  force- multiplying  efficiencies  to  assist  in  bridging  the  resource/demand 
gap.  One  such  efficiency  being  pursued  is  simultaneous  control  of  multiple  aircraft  by 
a  single  pilot,  or  Multi  Aircraft  Control  (MAC).  This  concept  of  operations  has  been 
documented  in  the  US  Air  Force  UAV  flight  Plan  (USAF,  2009),  which  calls  for  future 
systems  in  which  a  single  pilot  will  simultaneously  control  multiple  RPA  to  enable 
increased  aerial  surveillance  without  increasing  pilot  manpower  requirements.  Previ¬ 
ous  research  on  the  cognitive  workload  experienced  by  pilots  during  MAC  indicated 
that  frequent,  unpredictable,  and  oftentimes  overwhelming  volumes  of  communica¬ 
tion  events  are  able  to  produce  unmanageable  levels  of  system  induced  workload  for 
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MAC  pilots  (Schneider  and  McGrogan,  2011).  To  further  investigate  this  identified 
problem,  our  study  makes  use  of  IMPRINT  Pro,  a  Multiple  Resource  Theory  (MRT) 
based  dynamic,  stochastic  simulation  to  analyze  impacts  to  cognitive  workload  by  a 
disciplined  communication  modality  reallocation  construct. 

2.2  Background 

In  the  RPA  domain,  communication  is  a  continuous  and  demanding  process. 
Crews  must  track  information  on  weather,  threats,  mission  tasking,  mission  coordi¬ 
nation,  target  coordination,  airspace  coordination,  fleet  management,  and  status  and 
location  of  any  friendly  units,  etc.  The  RPA  pilot  is  not  only  responsible  for  aircraft 
control  but  is  also  a  critical  member  in  a  multi-path  communications  infrastructure 
(MITRE,  2009).  In  the  ground  station,  communication  with  the  pilot  takes  place  in 
one  of  two  modalities:  textual  chat  window(s)  or  the  speech-based  radio  systems.  At 
any  given  moment,  a  pilot  may  need  to  monitor  multiple  chat  windows  and  listen  to 
numerous  parties  operate  over  the  radio.  The  multitude  of  communication  sources 
and  different  media  coupled  with  the  quick  inter-arrival  rate  of  these  events  during 
a  dynamic  scenario  drives  an  incredible  cognitive  workload  for  the  pilot.  Cognitive 
or  mental  workload  expresses  the  task  demands  placed  on  an  operator  (Beevis  et  al., 
1999).  Task  demand,  or  task  load,  often  considers  the  goals  to  be  achieved  by  the  op¬ 
erator,  the  time  available  to  perform  the  tasks  necessary  to  accomplish  the  goals,  and 
the  performance  level  of  the  operator  (Hardman  et  ah,  2008).  Therefore,  workload 
increases  when  the  number  or  difficulty  of  tasks  necessary  to  perform  a  goal  increase, 
or  when  the  times  allotted  to  complete  these  tasks  decrease.  Assuming  that  the  op¬ 
erator  has  a  given  amount  of  mental  resources  (e.g.,  attention,  memory,  etc.)  that  he 
or  she  can  utilize  to  complete  the  necessary  tasks,  mental  workload  corresponds  to 
the  proportion  of  the  operators  mental  resources  demanded  by  a  task  or  set  of  tasks. 
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Several  methods  have  been  employed  to  measure  and  quantify  mental  workload  over 
the  past  four  decades  and  have  been  summarized  in  numerous  publications  (Beevis 
et  al.,  1999;  Gawron,  2008;  Hardman  et  al.,  2008).  The  current  analysis  incorporates 
Multi  Resource  Theory  (MRT)  into  the  workload  calculations  to  account  for  channel 
conflict  driven  workload.  As  a  theory,  MRT  purports  the  existence  of  four  mental 
dimensions  (or  channels)  available  to  process  information  and  perform  tasks.  The 
channels  include  the  stages  of  processing  dimension,  the  codes  of  processing  dimen¬ 
sion,  the  modalities  dimension  and  the  visual  channels  dimension.  These  channels  are 
allocated  to  concurrent  tasks  with  the  difficulty  of  the  tasks  and  the  demand  conflict 
between  channels  driving  the  overall  mental  workload  value  (Wickens,  2008).  MRT 
falls  in  line  with  the  concurrent  nature  of  tasks  imposed  on  an  RPA  pilot  (performing 
primary  tasks  while  communicating  and  monitoring  communication)  and  is  therefore 
an  appropriate  theory  to  apply  to  the  present  analysis. 

2.3  Method 

Having  discussed  communication  events  and  the  incorporation  of  MRT,  it  can 
be  seen  that  the  specific  channels  employed  by  the  modeled  communication  events 
will  be  highly  relevant  to  the  MRT  workload  calculations.  As  communication  events 
begin  to  conflict  with  existing  work  activities  on  the  various  channels,  the  calculated 
overall  cognitive  workload  will  account  for  such  conflicts.  This  construct  postures 
the  analysis  to  be  able  to  address  the  question  of  whether  or  not  adjusting  the  in¬ 
tentional  allocation  of  communication  events  to  particular  modalities  will  be  able  to 
meaningfully  affect  overall  cognitive  workload. 
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2.3.1  Model. 


A  previous  model  of  pilot  mental  workload  Schneider  and  McGrogan  (2011)  was 
utilized  to  understand  the  impact  of  communications  modality.  This  model  employed 
functional  analysis  and  task  allocation  to  construct  an  executable  architecture  of  the 
multiple  RPA  system.  This  architecture  was  then  replicated  within  the  Improved 
Performance  Research  Integration  Tool  (IMPRINT)  to  estimate  the  pilots  workload 
under  various  mission  segments,  such  as  handover,  transit,  emergency,  benign  and 
dynamic  surveillance,  etc.  This  model  relied  on  Subject  Matter  Expert  (SME)  input 
to  develop  distributions  for  the  length,  frequency,  and  difficulty  of  the  events  that 
induce  workload  on  the  pilot.  The  original  research  on  this  model  indicated  that 
workload  was  particularly  high  during  what  were  termed  dynamic  mission  segments. 
These  mission  segments  often  involve  high  levels  of  communication  between  the  pilot 
and  external  actors  to  facilitate  the  tracking  or  observation  of  moving  targets.  High 
levels  of  communication  resulted  in  particularly  high  pilot  workload  while  operating 
a  single  aircraft  and,  excessive  workload  while  controlling  multiple  dynamic-mission 
aircraft.  The  original  research  indicated  that  a  reduction  in  pilot  workload  imposed 
by  communication  would  be  necessary  to  facilitate  MAC.  To  understand  the  poten¬ 
tial  impact  of  communication  modality  on  operator  workload,  the  communications 
portion  of  the  earlier  workload  model  was  modified  to  permit  communications  events 
to  be  reallocated  to  alternate  communications  modalities.  The  revised  model  per¬ 
mits  communication  events  that  were  originally  allocated  to  the  auditory  channels 
where  the  operator  listens  and  speaks  to  the  visual  and  fine  motor  channels  where 
the  operator  reads  and  types,  or  vice  versa. 

Figure  1  depicts  the  high  level  structure  of  the  revised  communications  model. 
The  gray  highlighted  elements  indicate  model  elements  that  were  added  to  facilitate 
this  particular  evaluation.  As  shown,  in  the  original  communication  model,  commu- 
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nication  events  were  generated  with  a  mission  segment  dependent  frequency.  As  a 
communication  event  was  generated,  it  was  assigned  as  either  an  auditory  event  or  a 
text-based  event  with  25%  of  the  events  being  allocated  as  auditory  events  and  the 
remaining  allocated  as  text  events.  Half  of  the  auditory  events  then  required  the  pilot 
to  talk  or  listen  while  90%  of  the  text  events  required  the  pilot  to  read  while  only  10% 
of  the  events  required  the  pilot  to  type  a  response.  Although  not  shown,  it  is  then 
assumed  that  some  percentage  of  the  final  events  generate  a  repeat  communication 
event,  indicative  of  a  continued  conversation. 

Figure  1.  Modified  Communication  Model  of  Pilot  Workload 


To  conduct  the  current  evaluation,  the  auditory  and  text  events  shown  in  gray 
have  the  potential  to  either  pass  an  auditory  or  text  event  as  a  respective  auditory 
or  text  event  or  to  convert  an  auditory  event  to  a  text  event  or  convert  a  text  event 
to  an  auditory  event.  With  this  modification,  it  is  assumed  that  the  characteristics 
of  the  communication  are  due  to  communication  needs,  such  that  if  a  text  event  in 
the  original  model  had  a  90%  chance  of  providing  an  input  to  the  pilot  and  only  a 


10%  chance  of  an  output  to  the  pilot,  a  text  event  converted  to  an  auditory  event 
has  a  90%  probability  to  require  the  pilot  to  listen  and  only  a  10%  probability  to 
require  the  pilot  to  talk.  The  parameters  V  (for  Voice  reallocation)  and  T  (for  Text 
reallocation)  provide  the  ability  to  convert  auditory  or  text  events  to  its  compliment. 
If  V  and  T  are  both  100%,  the  revised  model  equates  to  the  original  model.  Reducing 
either  of  these  parameters  permits  a  portion  of  one  type  of  communication  event  to 
the  converted  to  the  complimentary  communication  event. 

2.3.2  Experimental  Design. 

For  this  paper,  a  total  of  six  “levels”  of  voice/text  allocation  were  selected  such 
that  the  percent  of  voice  communication  were  varied  between  0  and  100  percent.  For 
levels  of  voice  communications  less  than  25%,  V  was  varied  while  T  was  maintained 
at  100%.  However,  for  levels  of  voice  communications  greater  than  25%,  V  was 
maintained  at  100%  while  T  was  varied  to  achieve  the  desired  communications  levels. 
All  analysis  was  performed  for  a  10  hour  dynamic  mission  segment  with  a  single 
pilot  operating  the  aircraft.  Although  IMPRINT  does  not  currently  have  built-in 
Monte  Carlo  functionality,  an  external  batch  application  was  developed  to  replicate 
numerous  runs.  A  total  of  10  replications  using  different  random  number  seeds  were 
computed  to  estimate  the  output  statistics.  The  output  of  the  IMPRINT  model  was 
analyzed  to  determine  the  proportion  of  time  that  the  operator  would  experience 
workload  values  over  a  specified  task  saturation  threshold.  A  workload  value  of  60 
was  calibrated  to  be  about  the  90%  of  operator  maximum  threshold,  which  indicates 
the  workload  value  a  pilot  can  experience  without  degraded  performance.  The  mean 
and  variance  across  the  10  replications  for  each  communication  ratio  was  calculated. 
Analysis  of  Variance  (ANOVA)  and  Tukey  post-hoc  tests  were  employed  determine 
the  statistical  differences  between  the  average  of  percent  time  over  threshold. 
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2.4  Results 


Figure  2  shows  the  percent  time  over  threshold  as  a  function  of  the  percentage  of 
voice  communication.  A  one  way  ANOVA  indicated  a  significant  effect  of  the  percent 
of  voice  communication  upon  the  percentage  of  time  over  threshold  (P  <  0.001).  As 
shown  in  Figure  1,  the  percent  of  time  over  threshold  is  reduced  as  the  percent  of 
voice  communication  is  increased  from  0%  to  40%.  At  40%  voice  communication  the 
percent  time  over  threshold  is  reduced  to  24.5%  compared  to  33.1%  with  0%  voice 
communication.  This  change  is  statistically  significant.  The  change  in  percent  time 
over  threshold  is  statistically  insignificant  as  the  percent  of  voice  communication  is 
increased  from  40%  to  60%.  This  trend  indicates  that  pilot  workload  is  reduced  by 
the  use  of  both  auditory  and  text-based  communications  in  this  system. 

Figure  2.  Percent  Time  Over  Threshold  as  the  percentage  of  voice  is  reallocated 


Results  further  show  that  the  percent  time  over  threshold  is  greater  at  0%  voice 
than  at  100%  voice  communications.  This  might  have  been  expected  as  reading 
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and  typing  likely  conflicted  directly  with  other  tasks  being  performed  by  the  pilot, 
including  visually  monitoring  the  status  and  manipulating  the  controls  of  the  RPAs. 
As  such  workload  is  highest  when  all  of  the  communication  is  allocated  entirely  to 
the  visual  channel. 

2.5  Conclusions 

The  model  indicates  that  by  deliberately  allocating  communication  between  au¬ 
ditory  and  text-based  modalities  the  pilots  workload  and  particularly  the  percent  of 
time  the  pilot  operates  above  their  task  saturation  threshold  can  be  statistically  re¬ 
duced.  The  model  shows  that  the  percent  of  time  over  threshold  is  greatest  when  all 
of  the  communication  is  allocated  to  the  text-based  communications  such  that  zero 
percent  of  the  communication  is  allocated  to  voice.  This  type  of  communication  is 
most  likely  to  conflict  with  other  tasks  involving  the  visual  system  to  monitor  the 
RPA  and  the  small  motor  system,  which  is  used  by  the  pilot  to  control  the  RPA. 
As  communication  events  are  moved  from  text  to  auditory,  the  workload  decreases. 
However,  as  more  communication  is  moved  to  the  auditory  channel,  the  percent  of 
mission  time  over  task  saturation  threshold  then  begins  to  increase.  The  increase 
likely  occurs  as  the  auditory  tasks  begin  to  overlap  and  conflict  with  one  another 
to  increase  workload.  There  appears  to  be  an  optimal  allocation  of  communications 
between  voice  and  text  modalities  to  achieve  the  lowest  workload  given  a  constant 
traffic  load.  Future  research  will  examine  dynamic  reallocation  of  modalities. 
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III.  Simulation  of  Distributed  Communication 


This  paper1  is  formatted  for  submission  to  the  International  Journal  of  Human 
Factors  Modelling  and  Simulation. 

Discrete  Event  Simulation  of  Real  Time  Human  Distributed  Team 

Communication 

Travis  Pond,  Michael  Miller,  John  Colombi,  Randall  Gibb 

Air  Force  Institute  of  Technology,  Wright-Patterson  Air  Force  Base,  Ohio,  45433, 
USA 

Abstract 

With  increasing  automation  of  Remotely  Piloted  Aircraft  (RPA),  the  capability 
for  a  single  operator  to  fly  multiple  vehicles  may  be  possible,  but  improved  Human 
Systems  Integration  becomes  important.  These  operations  rely  heavily  on  distributed 
team  communications  determined  by  the  systems  architecture.  This  research  investi¬ 
gates  the  effects  of  systems  architecture  on  operator  workload.  The  Improved  Perfor¬ 
mance  Research  Integration  Tool  (IMPRINT)  was  used  to  estimate  workload  using 
Multiple  Resource  Theory  for  four  system  configurations  simulated  in  four  mission  ac¬ 
tivity  levels.  Mental  workload,  monitoring  time  and  the  number  of  delayed  tasks  were 
estimated  to  determine  the  effect  of  changing  the  number  of  communication  nodes 
and  greater  automation  of  navigation  tasks.  Operator  interviews  provided  stochastic 
parameters  for  the  model.  The  analysis  demonstrates  that  removing  the  navigation 
tasks  has  a  greater  effect  on  task  delay  and  time  spent  building  situation  awareness 
(SA)  than  changing  the  number  of  communication  nodes  during  high  load  mission 
segments.  Changing  the  number  of  communication  nodes  has  a  greater  effect  on  men¬ 
tal  workload  than  exclusion  of  the  navigation  task.  System  architecture  changes  have 
1This  paper  was  written  in  UK  English 
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little  effect  for  mission  segments  with  low  load.  This  research  has  implications  for 
future  design  of  RPA  or  distributed  team  systems. 

3.1  Introduction 

The  demand  for  increasingly  complex  medium  altitude  remotely  piloted  aircraft 
(RPA)  platforms  and  their  maturing  capabilities  has  brought  with  it  an  increased 
reliance  upon  distributed  team  communications.  Many  U.S.  government  organiza¬ 
tions,  such  as  the  military  services,  Border  Patrol,  NASA  (NASA,  2011)  and  the 
Department  of  Homeland  Security,  have  requirements  for  RPA  technology  and  have 
an  expressed  interest  in  expanding  their  respective  fleets  of  RPAs.  Civil  uses  for  RPAs 
abound  (U.S.  Government  Accountability  Office,  2008).  However,  real  time  commu¬ 
nication  can  be  burdensome  in  intense  situations,  and  it  can  affect  the  ability  of  the 
pilot  and  sensor  operator  to  perform  their  respective  primary  tasks.  Offloading  of 
communication  can  keep  the  RPA  operators  from  becoming  task  saturated  (Wickens 
et  ah,  2003).  Communication  offloading  is  accomplished  in  current  U.S.  Air  Force 
(USAF)  operations  by  the  addition  of  mission  intelligence  coordinators  (MIC)  who 
serve  to  facilitate  external  communications.  This  research  uses  a  cognitive  model  to 
predict  MIC  workload  and  inform  future  communications  architecture  development 
for  remotely  piloted  systems. 

In  the  USAF,  network  technologies  in  RPA  systems  have  increased  the  number  of 
communication  tasks  that  operators  must  perform.  As  a  result,  operators  communi¬ 
cate  with  more  parties  more  often  than  with  previous  radio  and  land-line  telephone 
technologies.  In  situations  where  both  high  volumes  of  communication  and  high  op¬ 
erator  workload  exist,  there  may  be  a  correlation  between  communication  and  work¬ 
load.  The  hypothesis  of  this  research  is  that  the  current  communications  architecture 
induces  high  task  demand,  and  resultant  workload  on  the  operator.  The  primary 
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research  question  is:  How  does  communication  architecture  affect  the  workload  expe¬ 
rienced  by  a  system  operator? 

A  military  example  was  examined  to  answer  this  question.  This  example  is  extensi¬ 
ble  to  future  civil  and  commercial  systems,  some  of  which  already  use  similar  systems. 
Implications  of  this  research  are  important  for  designing  new  systems  which  require 
extensive  communication  between  distributed  teams  requiring  real-time  communica¬ 
tion.  Real-time  distributed  communication  is  a  cornerstone  of  military  strength  and  a 
boon  to  many  organizations  who  use  or  plan  to  use  RPAs;  this  research  addresses  ways 
in  which  communication  architecture  could  be  improved  to  allow  for  more  effective 
real  time  distributed  communication. 

3.2  Background 

The  background  of  this  research  combines  literature  from  several  otherwise  dis¬ 
parate  fields  of  study  in  an  attempt  to  bridge  the  gaps  between  them  and  provide  a 
solid  foundation  for  analysis.  First,  the  premise  of  workload  modelling  and  its  rele¬ 
vance  to  systems  design  will  be  discussed.  Background  information  about  the  military 
example  will  then  be  given  along  with  relevant  architecture  diagrams. 

Mental  workload  is  the  characterization  of  limited  human  mental  resource  de¬ 
mands  (Cain,  2007;  Wickens  and  Yeh,  1986).  For  the  purposes  of  this  research,  men¬ 
tal  workload  is  a  unit-less  and  relative  measure  derived  from  the  combination  of  tasks 
imposed  on  the  human,  their  respective  mental  resource  demands  and  the  degree  to 
which  the  tasks  place  demands  on  conflicting  or  complimentary  cognitive  channels 
(Wickens  and  Yeh,  1986;  Wickens,  2008).  Multiple  resource  theory  (MRT)  is  the 
concept  that  human  cognitive  resources  are  divided  into  multiple  attentional  ‘pools’, 
which  are  taxed  differently  depending  on  task  load.  Studies  showing  the  extent  to 
which  time  sharing  or  multitasking  situations  use  different  cognitive  processing  struc- 
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tures  provide  scientific  evidence  for  the  validity  for  this  theory.  A  multi-dimensional 
model  of  MRT  is  given  in  (Wickens,  2008).  The  model  is  essentially  an  input-process- 
output  model  of  human  perception,  cognition  and  action  with  resource  distinctions 
drawn  along  modal  lines.  The  codes  and  modalities  of  processing  distinguish  among 
resources  used  for  sensory  input  and  working  memory.  The  codes  of  processing  are 
spatial  and  verbal,  while  the  modalities  are  auditory  and  visual.  The  codes  and 
modalities  represent  the  combinations  in  which  humans  perceive  information.  They 
are  spatial-auditory,  spatial-visual,  verbal-auditory  and  verbal-visual.  The  stages 
of  processing  represent  the  modalities  of  cognitive  process  resources  which  are  used 
to  select  and  execute  action,  and  the  responses  are  manual-spatial  or  vocal-verbal. 
This  study  examined  operator  workload  for  verbal  codes  of  processing,  both  modal¬ 
ities,  and  vocal-verbal  responses.  For  example,  when  an  operator  receives  and  sends 
chat  messages,  she  perceives  them  using  verbal-visual  resources,  processes  them  using 
vocal- verbal  resources,  and  responds  using  vocal-verbal  resources. 

The  method  used  in  this  research  to  predict  mental  workload  using  MRT  and  dis¬ 
crete  event  simulation  is  found  in  Keller  et  al.  (2002).  Schneider  and  McGrogan  (2011) 
used  the  Improved  Performance  Research  Integrated  Tool  (IMPRINT)  to  implement 
the  method  described  in  Keller  et  al.  (2002)  to  predict  RPA  pilot  mental  workload. 
Workload  predictions  were  used  to  understand  the  manpower  implications  of  multiple 
aircraft  control.  To  develop  the  mental  model  of  the  pilot,  Schneider  and  McGrogan 
created  executable  architecture  to  translate  the  system  design  into  a  dynamic  model. 
Wang  and  Dagli  (2008,  pg.  1)  notes  that  “Architecture  modeling  furnishes  abstrac¬ 
tions  for  use  in  managing  complexities,  allowing  engineers  to  visualise  the  proposed 
system  and  to  analyze  the  problem  domain  and  describe  and  specify  the  architecture 
for  the  solution  domain.”  Architecture  modelling  methods  were  used  by  Schneider 
and  McGrogan  and  Mitchell  (2000).  Schneider  and  McGrogan’s  findings  suggest  that 
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communication  is  a  major  source  of  workload  for  RPA  operators.  Mitchell  found 
that  fMPRINT  predictions  of  communication  times  and  frequencies  correlated  with 
recorded  communications  amongst  a  platoon  of  soldiers  during  a  simulated  mission. 

The  architecture  modelled  in  this  research  is  that  of  U.S.  RPA  communications, 
centered  around  the  MIC,  who  is  part  of  a  distributed  team  coordinating  to  complete 
remote  surveillance  missions.  According  to  subject  matter  experts  (SME)  there  are, 
during  any  given  mission,  between  10  and  15  external  organizations  with  which  the 
MIC  must  communicate  regularly;  each  of  these  constitutes  one  network  ‘node’.  The 
RPA  crew  includes  a  pilot,  a  sensor  operator,  and  a  supervisor.  The  MIC  provides  a 
communication  buffer  between  the  crew  and  the  other  organizations  in  the  commu¬ 
nication  network.  A  diagram  of  representative  nodes  in  the  system  is  given  in  Figure 
3.  The  MIC  works  in  the  RPA  Operations  (Ops)  Center.  The  connections  between 
external  nodes,  the  pilot,  and  sensor  operator  are  not  shown. 

Figure  3.  Diagram  of  System  Nodes 


Selection  of  an  appropriate  modelling  tool  that  supported  calculation  of  human 
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mental  workload  was  crucial  to  the  understanding  architectural  effects  on  communi¬ 
cation  induced  workload.  IMPRINT2  was  selected  because  of  its  extensive  human 
performance  modelling  capability.  An  executable  system  architecture  model  can  be 
built  within  IMPRINT  using  a  functional  decomposition  to  describe  task  times  and 
resource  demands.  IMPRINT  provides  a  common  ground  between  executable  sys¬ 
tems  architecture  and  analytical  workload  modelling.  IMRPINT  was  developed  by 
the  US  Army  Research  Laboratories  to  support  manpower  and  personnel  systems 
analysis,  but  provides  the  ability  to  estimate  human  mental  workload.  This  tool  is  a 
dynamic,  stochastic,  discrete  event  network  modeling  tool  based  on  the  Micro  Saint 
Sharp  modeling  language  (USARL,  2010). 

The  creation  of  an  IMPRINT  model  requires  the  functional  decomposition  and 
allocation  of  tasks  to  users  or  crew  members.  Workload  values  are  then  assigned, 
following  the  process  described  in  Keller  et  al.  (2002),  to  each  resource/interface  pair 
and  then  to  each  task  performed  by  the  user.  Mental  workload  values  are  calcu¬ 
lated  based  upon  multiple  resource  theory  (MRT),  which  permits  the  modeling  of  the 
effect  of  both  sequential  and  concurrent  tasks  upon  human  mental  workload  (Woj- 
ciechowski,  2006;  Mitchell,  2003).  As  the  model  runs,  single  task  demand  values  are 
calculated  for  each  task  performed  by  the  user.  When  multiple  tasks  are  provided  to 
the  user  concurrently,  competing  for  limited  attentional  resources,  conflict  values  are 
determined  based  upon  the  user’s  ability  to  perform  these  concurrent  tasks.  Task  de¬ 
mands  which  lead  to  conflict  produce  nonlinear  increases  in  system  induced  workload 
as  more  tasks  are  added.  The  algorithm  which  IMPRINT  uses  to  calculate  mental 
workload  is  a  combination  of  a  task-resource  assignment  method  (Keller  et  ah,  2002) 
and  a  complex,  dynamic  set  of  summations  (Mitchell,  2000). 

To  construct  an  IMPRINT  model,  users,  tasks,  task  arrangement,  task  times, 
2Version  3.1.0.86 


17 


task  difficulties,  and  likely  errors  must  be  determined.  Each  of  these  model  elements 
can  be  deterministic  or  stochastic  in  nature.  The  discrete  events  in  the  model  then 
represent  real  events  and  permit  a  real-time  trace  of  system-induced  workload.  One 
of  the  important  features  of  IMPRINT  is  the  behavior  of  the  model  during  times 
when  the  operator  is  task  saturated  and  unable  to  simultaneously  address  all  of  the 
overlapping  tasks  that  are  allocated.  IMPRINT  provides  many  options,  including 
computing  workload  with  the  assumption  that  the  operator  will  complete  all  tasks 
even  when  the  workload  is  beyond  their  ability.  Workload  mitigation  strategies  may 
also  be  modelled,  representing  ways  in  which  real  users  might  allay  the  effects  of  task 
overlap  and  overload.  There  are  four  strategies  available.  When  a  new  task  in  the 
task  queue  would  cause  workload  to  rise  above  a  preset  threshold,  the  user  may  be 
modelled  as  offloading  the  new  task  to  a  contingent  operator,  delaying  the  new  task, 
dropping  the  new  task  from  the  queue  completely,  or  interrupting  the  current  task  to 
perform  the  new  task  and  completing  the  first  task  in  a  window  of  opportunity. 

3.3  Methodology 

Four  MICs  were  interviewed  to  gather  input  data  for  the  mental  workload  model. 
All  four  MICs  were  operationally  qualified,  2  were  enlisted  and  2  were  officers,  each 
with  more  than  two  years  of  operational  experience.  The  first  part  of  the  interview 
addressed  the  logical  architecture  of  the  system.  The  questions  were  designed  to 
give  insight  into  the  MICs’  primary  goals,  major  functions,  and  the  order  in  which 
tasks  are  performed.  These  questions  also  elicited  the  relationships  between  major 
tasks  and  external  events.  The  second  part  of  the  interview  addressed  specific  tasks, 
their  respective  durations  and  their  respective  difficulties  and  derivative  tasks.  The 
interview  questions  were  designed  to  depict  all  tasks  which  the  MIC  performs  during 
a  typical  shift. 
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Data  from  this  effort  indicated  that  during  a  typical  shift,  the  MIC’s  primary 
goal  is  to  perform  the  tasks  which  are  assigned  in  the  pre-mission  brief.  Typically, 
these  include  posting  way-point  coordinates  in  a  navigation  system  visible  to  nearly 
all  operators  in  the  area,  relaying  important  information  between  the  pilot  and  the 
other  nodes,  referencing  the  mission  plan,  and  acting  as  an  auxiliary  monitor  of  the 
full  motion  video  (FMV)  feed  from  the  RPA.  The  MIC  also  maintains  the  mission 
report  for  the  entire  mission  and  updates  it  between  tasks. 

3.3.1  Choice  of  Dependent  and  Independent  Variables. 

Three  dependent,  two  independent,  and  one  system  variable  were  selected  to  define 
the  model.  Mental  workload,  monitoring  time  and  the  number  of  delayed  events 
are  the  dependent  variables  while  the  number  of  external  communication  nodes  and 
exclusion  of  navigation  tasks  are  independent  variables.  The  arrival  rate  of  units  of 
activity  is  a  system  variable  used  to  compare  the  effects  of  the  independent  variables 
across  four  mission  task  loading  scenarios. 

The  mean  exponential  interarrival  time  of  major  events,  A maj,  was  assigned  four 
levels:  30,  60,  120,  and  1200  seconds  to  represent  the  variable  and  unpredictable 
nature  of  real  missions.  SMEs  described  missions  ranging  from  overwhelming  task 
loads  (represented  by  \maj  =  30)  to  nearly  abject  boredom  (represented  by  Xmaj  = 
1200).  The  levels  between  were  selected  arbitrarily  to  provide  more  resolution  near 
the  busy  end  of  the  scale.  The  number  of  communication  nodes  with  which  the  MIC 
must  communicate  is  represented  by  ncomm ,  and  the  control  value  of  ten  is  derived 
from  Figure  3.  The  navigation  task  parameter,  Nav  has  two  levels,  on  and  off,  to 
represent  the  inclusion  (on)  or  exclusion  (off)  of  the  navigation  tasks  in  the  model. 
The  simulation  length,  TE,  was  set  to  7200  seconds,  or  two  hours.  Two  hours  was 
selected  as  the  length  at  which  the  system  exhibited  stable  behavior,  and  variance 
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between  simulation  runs  was  very  low  for  workload  observations. 

Mental  workload  was  selected  as  a  dependent  variable  because  of  its  relevance 
to  system  design.  The  system  modelled  in  this  research  primarily  requires  cognitive 
work  more  than  physical  work.  Mental  workload  values  are  calculated  using  difficulty 
ratings  for  individual  tasks,  the  cognitive  resources  which  those  tasks  require,  the 
presence  of  other  tasks,  and  how  much  the  required  resources  conflict  between  the 
multiple  tasks.  Delayed  task  count  is  a  count  of  all  the  tasks  which  needed  to  be 
delayed  by  the  operator  because  workload  was  over  a  certain  threshold,  or  ‘red  line’. 
IMPRINT  allows  users  to  model  both  system  induced  workload  with  no  notion  of 
the  operator  and  the  workload  experienced  by  the  operator.  All  of  the  conditions 
described  in  the  design  of  experiments  were  run  again  with  a  workload  management 
strategy,  which  simulates  an  operator  present  who  would  “[p]erform  tasks  sequentially, 
beginning  with  ongoing  tasks  and  then  performing  the  next  task”  (Mitchell,  2000; 
Alion  Science  and  Technology,  2011).  The  delayed  task  count  shows  how  many  of  the 
tasks  during  the  mission  might  have  been  delayed  because  of  high  workload. 

It  is  assumed  that  the  operator  will  monitor  up  to  six  computer  monitors  to  detect 
changes  in  information  displayed  through  two  networked  computer  systems  during 
any  available  time  between  other  tasks.  Monitoring  time  is  the  sum  of  the  durations 
of  monitoring  tasks  over  the  course  of  the  two  hour  mission  segment.  Specifically, 
the  monitoring  tasks  are  composed  of  monitoring  a  FMV  feed,  a  navigation  system, 
and  up  to  15  IRC  windows.  Monitoring  the  navigation  system  is  considered  both  a 
navigation  task  and  a  monitoring  task  for  the  purposes  of  this  research.  Monitoring 
time  represents  the  ability  of  the  MIC  to  complete  the  secondary  task  of  maintaining 
situation  awareness  (SA).  SA  is  defined  as  the  perception  and  understanding  of  one’s 
environment  and  the  ability  to  predict  the  status  of  the  environment  in  the  near 
future  (ENDSLEY,  1995;  Tsang  and  Vidulich,  2002).  The  MICs  ability  to  maintain 
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an  accurate  picture  of  the  operational  environment  is  very  important  to  the  other 
members  of  the  team;  system  monitoring  time  is  a  measure  of  the  ability  to  allocate 
time  to  this  task. 


3.3.2  Experimental  Design. 

The  analysis  follows  a  fractional  factorial  design  where  each  level  of  ncomm  and 
Nav  are  compared  at  each  level  of  A maj  for  each  of  the  three  metrics.  For  a  full 
analysis  of  the  effects  of  the  independent  variables,  sixteen  mission  conditions  were 
modelled.  Each  of  these  sixteen  conditions  was  run  five  times  with  workload  man¬ 
agement  strategies  ‘on’  and  another  five  times  with  workload  managements  strategies 
‘off’.  The  presence  or  absence  of  workload  management  strategies  was  not  considered 
an  independent  variable;  replications  with  the  strategies  employed  were  used  to  count 
the  number  of  delayed  tasks.  The  experimental  conditions  are  listed  in  Table  1. 

Table  1.  List  of  Conditions 


Condition  ID 

^ maj 

^ comm 

Nav.  Task 

1 

30 

10 

on 

2 

30 

10 

off 

3 

30 

6 

on 

4 

30 

6 

off 

5 

60 

10 

on 

6 

60 

10 

off 

7 

60 

6 

on 

8 

60 

6 

off 

9 

120 

10 

on 

10 

120 

10 

off 

11 

120 

6 

on 

12 

120 

6 

off 

13 

1200 

10 

on 

14 

1200 

10 

off 

15 

1200 

6 

on 

16 

1200 

6 

off 
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The  number  of  replications  was  determined  by  following  the  process  outlined  in 
Banks  et  al.  (2010).  Using  condition  1,  which  was  thought  to  be  the  most  variable, 
the  desired  half-widths  of  the  confidence  intervals  were  obtained  by  completing  five 
replications  for  each  of  the  sixteen  conditions,  resulting  in  160  independent  repli¬ 
cations.  Results  were  calculated  for  five  random  number  seeds  between  1  and  1000 
using  a  random  integer  generator  function  in  MATLAB.  The  half  widths  were  deemed 
sufficient  if  they  included  no  more  than  ±5%  of  the  mean. 

Data  groups  were  compared  for  purposes  of  determining  meaningful  effects.  Of 
the  large  set  of  possible  pairwise  comparisons,  only  sixteen  were  made  to  determine 
the  significance  of  the  four  system  configurations.  The  two  control  groups,  where 
nComm  =  10  and  Nav  =  ‘On’  were  compared  pairwise  with  the  alternate  groups, 
where  ncomm  =  6  and  Nav  =  ‘Off’.  These  four  comparisons  were  made  at  each  of  the 
four  levels  of  A maj- 

3.4  Model  Description 

3.4.1  Assumptions. 

It  is  assumed  that  a  ‘unit’  of  activity,  such  as  the  observation  of  a  single  suspicious 
action  performed  by  the  surveillance  target,  constitutes  a  major  event  and  causes  the 
MIC  to  perform  communications  once  for  each  node  in  the  external  system.  The 
performance  of  one  communication  task,  e.g.  internet  relay  chat  (IRC)  or  intercom 
use,  constitutes  a  minor  event.  The  assumption  that  one  minor  event  occurs  for 
each  node  for  each  major  event  is  valid  for  the  following  reasons:  1)  it  was  reported 
by  SMEs  that  all  of  the  mission  related  communication  occurs  because  of  external 
events,  2)  interviews  revealed  that  during  high  communication  situations,  MICs  find 
themselves  communicating  ‘constantly’,  presumably  more  often  than  once  per  node 
per  external  event.  Therefore,  the  estimate  of  one  communication  per  node  per  major 


22 


event  is  a  valid  assumption.  The  model  assumes  that  each  major  event  triggers  one 
minor  communication  event  for  each  node. 

Another  important  assumption  is  that  the  system  would  demand  that  the  MICs 
handle  these  communication  events  sequentially  and  not  simultaneously.  Interviews 
revealed  that  when  MICs  would  communicate  with  some  nodes,  they  would  some¬ 
times  not  receive  a  response  for  up  to  five  minutes,  meaning  that  for  most  of  their 
communications,  response  time  is  not  critical  within  a  reasonable  window.  The  in¬ 
terviewees  also  noted,  however,  that  they  communicated  constantly  during  major 
events.  The  duration  of  major  events  is  increased  or  decreased  by  adjusting  the  ex¬ 
ponentially  distributed  interarrival  times  of  minor  events,  A comm,  which  represents 
the  rate  at  which  the  system  requires  the  MIC  to  complete  communication  tasks. 
It  was  also  reported  that  for  a  single  unit  of  activity,  it  took  approximately  twenty 
seconds  for  the  communications  to  stop.  To  accurately  model  the  interarrival  of  mi¬ 
nor  communication  events,  the  author  modelled  a  single  major  event  and  ten  nodes. 
Experimentation  with  the  interarrival  time  of  minor  communication  events  led  the 
author  to  an  assumed  exponential  arrival  rate  of  one  every  three  seconds  to  achieve 
the  twenty  second  duration  of  major  events.  Therefore,  A comm  is  set  to  three  seconds 
for  the  purpose  of  this  research. 

The  model  includes  only  the  lower  level  tasks  which  were  determined  to  be  critical 
to  the  communication  functions  of  the  MIC.  The  tasks  included  in  the  model  are 
three  monitoring  tasks,  three  sequential  tasks  related  to  posting  coordinates  into  the 
navigation  system,  as  well  as  reading,  typing,  listening  and  speaking  both  complex 
and  simple  sentences.  The  model  has  represented  within  it  fourteen  total  MIC  tasks. 
The  tasks  are  arranged  within  the  model  according  to  the  information  given  in  the 
interviews  and  in  existing  architecture.  Task  durations  and  difficulties  were  assigned 
conservatively  using  the  visual,  auditory,  cognitive  and  psychomotor  (VACP)  rating 


23 


scales  within  IMPRINT  (Keller  et  ah,  2002;  Alion  Science  and  Technology,  2011). 


3.4.2  Structure  and  Logic. 

There  are  two  types  of  tasks  in  the  model,  automated  tasks  which  are  not  included 
in  the  workload  calculation,  and  operator  tasks,  which  do  carry  workload  values.  The 
model  boundaries  are  defined  by  the  operator  tasks,  which  are  detailed  in  Table  2, 
where  normal  distribution  values  are  given  as  /i,  a  and  triangular  values  are  given 
as  mode,  min,  max.  The  structural  tasks  represent  the  system  which  is  external  to 
the  model  boundary.  Only  tasks  performed  by  the  MIC  are  included  explicitly,  while 
external  stimuli,  which  may  represent  the  actions  of  other  operators,  are  abstracted 
to  the  tasks  which  they  induce  the  MIC  to  perform.  These  external  stimuli  are 
represented  as  entity  generating  ‘dummy’  tasks  which  generate  MIC  tasks  according 
to  the  logic  stated  above.  On  its  foundational  structural  level,  the  model  is  a  queueing 
system  where  entities  arrive  with  exponentially  distributed  rates,  and  are  ‘processed’ 
by  tasks  with  normally  distributed  processing  times.  These  entities  represent  the 
external  events.  Network  edges  or  stochastic  logical  code  executed  during  the  tasks 
determines  the  entities’  next  destinations. 
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Table  2.  MIC  Low-Level  Tasks 


Task 

Duration  Distribution 

Duration  (seconds) 

Single  Task  Demand 

Monitor  Navigation  System 

Triangular 

6,  4,  9 

5.0 

Monitor  FMV 

Triangular 

10,  3,  21 

6.0 

Monitor  mIRC 

Triangular 

6,  4,  9 

5.0 

Copy  Coordinates  From  mIRC 

Normal 

1.0,  0.25 

8.5 

Find  Navigation  System  Monitor 

Normal 

1.0,  0.25 

4.0 

Paste  Coordinates  into  the  system 

Normal 

1.0,  0.25 

8.5 

Read  Simple  Sentence 

Normal 

2.0,  0.5 

5.1 

Type  Simple  Sentence 

Normal 

1.5,  0.5 

7.0 

Listen  to  Simple  Sentence 

Normal 

2.0,  0.5 

3.0 

Speak  Simple  Sentence 

Normal 

1.5,  0.5 

2.0 

Read  Complex  Sentence 

Normal 

7.0,  2.0 

5.1 

Type  Complex  Sentence 

Normal 

6.0,  2.0 

7.0 

Listen  to  Complex  Sentence 

Normal 

7.0,  4.0 

6.0 

Speak  Complex  Sentence 

Normal 

5.0,  2.0 

4.0 

3.5  Analysis  &  Results 

3.5.1  Methods  of  analysis. 

Data  for  each  metric  was  tested  for  normality;  comparison  groups  were  tested  for 
significant  differences.  All  confidence  intervals  are  calculated  at  a  =  0.05.  Lillicfors 
tests  were  conducted  on  replication  means  within  groups  to  determine  normality.  3 
Three-way  analysis  of  variance  (AN OVA)  tests  were  performed  on  group  means  to 
determine  which  changes  had  significant  effects  on  the  dependent  variables.  Post-hoc 
tests  were  conducted  for  added  specificity  about  the  differences  between  conditions. 

Distribution  of  Workload  Data.  For  a  given  replication,  the  weighted 
mean  of  workload  values  where  Wi  represents  workload  and  di  represents  the  duration 

3The  Lilliefors  test  was  used  because  it  is  valid  for  small  sample  sizes,  where  the  chi-square  test 
is  not  as  useful  (Lilliefors,  1967).  Also,  the  Lilliefors  test  is  non-parametric,  thus  the  parameters  for 
the  normal  distribution  against  which  the  sample  is  tested  do  not  need  to  be  given  as  they  would 
be  for  the  one-sample  Kolmogorov-Snrirnov  (KS)  test. 
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at  that  workload  value  is  computed  by  Eq.  1. 


w  = 


E  wi  •  dj 

Ed* 


(i) 


Lillefors  tests  were  performed  on  each  condition  to  test  for  normality.  Data  which 
is  normally  distributed  is  considered  to  be  the  result  of  a  random  process,  and  can 
statistically  tested  as  such.  The  approach  used  in  Eq.  1  to  estimate  the  means  was 
conducted  with  the  understanding  that  each  shape  is  different  by  design.  In  other 
words,  because  the  groups  are  being  compared  at  levels  of  A maj  each  group  has  a 
similar  set  of  workload  spikes  and  flat  periods  where  low  workload  occur,  though  they 
occur  at  different  times  during  the  mission  within  each  group.  The  replications  within 
each  group  are  similar  enough  that  differences  between  groups  are  easily  seen  from 
ANOVA  tests,  but  there  is  some  doubt  cast  on  these  tests  because  of  the  insensitivity 
of  weighted  averages  to  equal  changes  in  area  under  the  workload  function.  Therefore, 
an  additional  test  was  warranted. 

To  determine  whether  there  were  differences  between  the  conditions,  two  rounds 
of  two-sample  (KS)  tests  were  conducted.  The  two-sample  KS  test  computes  the 
distance  between  the  empirical  distribution  functions  (CDF)  of  the  two  samples. 
Because  two  empirical  CDFs  are  being  compared,  the  two-sample  KS  test  is  non- 
parametric.  The  null  hypothesis  for  the  two-sample  KS  test  is  that  the  two  samples 
come  from  the  same  continuous  distribution.  The  null  hypothesis  is  rejected  if  the 
distance  between  the  two  functions  is  sufficiently  large  (Darling,  1957).  Within  each 
group,  the  replications  were  compared  pairwise  to  determine  whether  each  replication 
within  the  group  came  from  the  same  continuous  distribution.  A  second  set  of  two- 
sample  (KS)  tests  was  performed  across  comparison  groups  to  determine  whether  the 
groups  came  from  different  continuous  distributions.  A  comparison  of  two  conditions 
is  shown  in  Figure  4. 
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Figure  4.  Workload  CDF  for  Two  Groups 


Workload  CDF  for  X  .  -  30,  Nav.  Task  =  On 


Monitoring  Time  &  Delayed  Tasks.  The  means  of  the  monitoring  times 
for  each  condition  were  normally  distributed  within  each  group.  Delayed  task  counts 
were  also  normally  distributed  within  each  group. 

3.5.2  Results. 

3. 5. 2.1  3  Way  Analysis  of  Variance  Tests. 

Each  of  the  three-way  AN OVA  tests  showed  that  the  largest  contributor  of  vari¬ 
ance  was  A maj.  This  was  expected;  A maj  changes  the  mission  scenario  and  the  essential 
shape  of  the  workload  function.  Only  the  effects  of  ncomm  and  Nav  are  discussed. 
In  each  case,  A maj  exhibited  two-way  interactions  with  the  other  two  variables.  The 
interaction  is  due  to  the  design  of  the  model;  decreasing  A maj  necessarily  decreases 
the  number  of  ncomm  events  and  the  number  of  times  the  Nav  task  was  executed 
in  the  Nav  =  ‘On’  cases.  In  other  words,  A maj  was  purposely  linked  to  the  other 
variables  in  an  indirect  way. 
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The  three-way  ANOVA  test  for  workload  showed  that  ncomm  had  a  significant  effect 
on  weighted  workload  averages  between  groups  (F  =  337.1,  P  <  0.0001).  Nav  was 
found  to  not  have  a  significant  effect.  There  was  a  small  three-way  interaction  between 
A maj ,  ncomm ,  and  Nav  (F  =  5.634,  P  <  0.0001).  For  monitoring  time,  the  three- 
way  ANOVA  test  showed  that  ncomm  had  a  significant  effect  between  groups  (F  = 
49.38,  P  <  0.0001).  Nav  was  found  to  have  a  much  greater  effect  (F  =  822.39,  P  < 
0.0001).  There  were  no  other  two  or  three-way  interactions.  The  three-way  ANOVA 
test  for  the  delayed  task  count  showed  that  ncomm  had  a  significant  effect  between 
groups  (F  =  93.59,  P  <  0.0001).  Nav  was  found  to  have  a  slightly  greater  effect 
(F  =  102.40,  P  <  0.0001).  There  were  no  other  two  or  three-way  interactions. 

3. 5. 2. 2  Workload. 

Results  for  workload  were  mixed.  KS  tests  within  groups  showed  that  several 
conditions  were  not  composed  of  replications  from  same  continuous  distributions. 
These  replications  were  removed  before  the  between  groups  KS  tests  were  conducted. 


Figure  5.  Weighted  Workload  Mean  Comparison  1 
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Weighted  Averages  Workload  Without  Navigation  Tasks 


comm  maj 


The  effect  was  greater  in  the  conditions  with  the  navigation  task  included.  Post- 
hoc  tests  showed  that  changing  levels  of  ncomm  was  only  significant  for  \maj  of  30,  60, 
and  120  seconds.  The  post-hoc  tests  showed  that  the  exclusion  of  the  tasks  was  only 
significant  in  one  comparison,  at  ncomm  =  10  and  A maj  =  30.  Figure  5  and  Figure  6 
show  how  changing  levels  of  ncomm  and  Nav  affect  weighted  mean  workload. 
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Figure  6.  Weighted  Workload  Mean  Comparison  2 
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3. 5. 2. 3  Monitoring  Time. 

Monitoring  time  results  showed  that  changing  levels  of  ncomm  had  a  greater  effect 
when  the  navigation  tasks  were  included  in  the  model.  This  is  partly  due  to  depen¬ 
dency  of  the  metric  on  the  measurement  of  these  tasks.  Changing  the  number  of 
communication  nodes  with  the  navigation  task  excluded  was  significant  in  only  one 
of  the  four  comparisons,  for  A maj  =  60.  The  effect  of  changing  ncomm  can  be  observed 
in  Figure  7. 


30 


Figure  7.  Monitoring  Time  Comparison  1 
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Post-hoc  tests  showed  that  excluding  the  navigation  tasks  was  significant  for  \maj 
of  30,  60,  and  120  seconds.  The  effect  of  excluding  the  navigation  tasks  can  be 
observed  in  Figure  8. 
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Figure  8.  Monitoring  Time  Comparison  2 
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3. 5. 2. 4  Delayed  Events. 

Post-hoc  tests  revealed  that  the  effect  of  reducing  the  number  of  communication 
nodes  on  the  number  of  delayed  events  was  significant  for  \maj  of  30  seconds  in  the 
Nav  =  ‘On’  condition.  In  the  Nav  =  ‘Off’  condition,  changing  the  level  of  ncomm  was 
significant  to  the  changes  in  mean  for  Xmaj  of  30,  60,  ad  120  seconds. 


32 


Figure  9.  Delayed  Task  Comparison  1 
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The  post-hoc  tests  also  showed  that  exclusion  of  the  navigation  tasks  had  a  sig¬ 
nificant  effect  on  the  number  of  delayed  tasks  for  both  levels  of  ncornrn  for  \maj  of  30 
and  60  seconds.  The  effect  of  reducing  the  number  of  communication  nodes  is  shown 
in  Figure  9,  while  the  effects  of  excluding  the  navigation  tasks  are  shown  in  Figure 
10. 
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Figure  10.  Delayed  Task  Comparison  2 
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A  multiple  comparison  of  means  post-hoc  procedure  was  conducted  using  the 
statistics  from  the  three-way  ANOVA  tests.  The  test  returns  a  Bonferroni  adjusted 
probability  that  the  means  are  not  significantly  different.  Percent  improvement  was 
calculated  using  results  from  the  post-hoc  tests  by  calculating  the  change  in  mean  for 
system  changes  that  resulted  in  statistically  significant  improvements  in  each  of  the 
three  metrics.  These  improvements  are  shown  in  Table  3.  Bold  type  signifies  values 
which  were  significant  in  post-hoc  tests. 
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Table  3.  System  Improvements 


Workload 

Monitoring  Time 

Delayed  Tasks 

Change 

Constant 

^ maj 

35.42% 

5.28% 

32.84% 

Change  Nodes  from  10  to  6 

Nav  On 

30 

24.41% 

3.96% 

22.53% 

Change  Nodes  from  10  to  6 

Nav  On 

60 

19.35% 

2.69% 

47.76% 

Change  Nodes  from  10  to  6 

Nav  On 

120 

2.58% 

0.00% 

50.46% 

Change  Nodes  from  10  to  6 

Nav  On 

1200 

26.18% 

2.49% 

38.79% 

Change  Nodes  from  10  to  6 

Nav  Off 

30 

27.35% 

2.72% 

42.01% 

Change  Nodes  from  10  to  6 

Nav  Off 

60 

25.94% 

1.68% 

53.45% 

Change  Nodes  from  10  to  6 

Nav  Off 

120 

10.36% 

0.33% 

18.57% 

Change  Nodes  from  10  to  6 

Nav  Off 

1200 

11.89% 

19.97% 

35.44% 

Change  Nav  from  On  to  Off 

Tlcomm  10 

30 

0.00% 

12.60% 

36.13% 

Change  Nav  from  On  to  Off 

W'comm  10 

60 

0.00% 

6.79% 

34.33% 

Change  Nav  from  On  to  Off 

W'comm  10 

120 

0.00% 

0.00% 

63.98% 

Change  Nav  from  On  to  Off 

Tlcomm  10 

1200 

0.00% 

16.79% 

41.16% 

Change  Nav  from  On  to  Off 

T^comm  6 

30 

0.00% 

11.54% 

48.27% 

Change  Nav  from  On  to  Off 

Tlcomm  6 

60 

3.32% 

5.75% 

41.47% 

Change  Nav  from  On  to  Off 

71  — fi 

120 

0.00% 

0.83% 

40.80% 

Change  Nav  from  On  to  Off 

W'comm  6 

1200 

Generally,  changing  the  system  architecture  resulted  in  greatest  improvement  dur¬ 
ing  periods  of  high  event  frequency  (A  =  30,  60  seconds).  Little  improvement  is  shown 
for  changing  the  system  during  periods  of  low  event  frequency  (A  =  120, 1200  sec¬ 
onds)  . 

3.6  Discussion 

3.6.1  Conclusions. 

Mental  workload  in  a  system  of  distributed  teams  can  be  lessened  at  the  sys¬ 
tem  level  by  reducing  the  number  of  network  nodes  with  which  the  operator  must 
communicate  frequently.  Mental  workload  can  be  decreased  to  a  lesser  extent  by 
decreasing  the  number  of  secondary  tasks  which  the  system  requires  the  operator  to 
perform.  Reducing  the  number  of  secondary  tasks  has  little  to  no  significant  effect 
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on  mental  workload  if  the  number  of  communication  nodes  is  already  low.  For  future 
systems,  therefore,  it  would  be  more  beneficial  to  focus  design  on  reducing  the  opera¬ 
tor’s  communication  nodes  rather  than  ensuring  that  the  operator  had  no  secondary 
tasks.  Additionally,  creating  an  additional  node  to  handle  more  communication  is 
most  useful  for  systems  which  generate  communication  events  with  high  frequency, 
in  this  case,  10  per  30  seconds. 

The  amount  of  time  which  the  operator  can  use  to  monitor  the  system  and  build 
valuable  situation  awareness  is  increased  effectively  by  reducing  the  number  of  com¬ 
munication  nodes  or  reducing  the  number  of  secondary  tasks.  If  the  secondary  tasks 
include  monitoring  tasks,  reducing  the  number  of  communication  nodes  has  little 
effect  if  these  tasks  are  already  excluded.  Reducing  the  number  of  communication 
nodes  and  exclusion  of  secondary  tasks  both  have  significant  effects  in  systems  with 
high  frequencies  of  communication-inducing  events.  Reducing  the  number  of  commu¬ 
nication  nodes  is  more  effective  if  there  are  no  secondary  tasks.  Reducing  the  number 
of  secondary  tasks  was  effective  for  systems  with  both  levels  of  communication  nodes 
at  the  two  higher  levels  of  communication-inducing  event  frequency,  implying  that 
reducing  secondary  tasks  reduces  the  probability  that  tasks  will  be  delayed. 

The  effective  number  of  communication  nodes  can  be  reduced  in  at  least  three 
ways.  This  research  assumes  that  a  second  operator  has  been  introduced  in  the 
ricomm  —  6  conditions,  and  has  taken  half  of  the  first  operators  communications 
nodes.  This  configuration  is  shown  in  Figure  11.  The  first  operator  loses  five  of 
the  original  nodes  and  adds  one  node,  which  is  the  second  operator,  giving  a  total 
of  six  nodes  to  each  operator,  in  which  case,  the  original  operator  is  modelled  as 
simply  having  four  fewer  nodes.  The  nodes  in  Figure  11  have  been  reallocated  such 
that  each  MIC  communicates  with  a  somewhat  related  set  of  nodes.  Tactical  and 
intelligence  gathering  organisations  are  connected  to  one  MIC,  while  air  traffic  con- 


36 


trol  and  authority  organisations  are  allocated  to  the  other.  These  allocations  would 
ideally  change  dynamically  to  avoid  one  MIC  being  overwhelmed  while  the  other  is 
bored.  This  allocation  organises  information  according  to  the  operator’s  goal,  and 
thus  promotes  situation  awareness  (Endsley  et  ah,  2003). 

Figure  11.  Alternate  Diagram  of  System  Nodes  With  Two  MICs 


Another  way  in  which  the  reduction  of  communication  nodes  can  be  achieved  is 
through  automation.  The  interviewees  reported  that  about  one  half  of  the  communi¬ 
cation  tasks  they  perform  involve  the  simple  passing  of  information  from  one  party  to 
another.  If  this  task  could  be  automated  using  software  which  filters  the  chat  stream 
and  passes  information  between  the  correct  parties,  the  effective  number  of  nodes 
would  be  reduced.  A  third  way  to  reduce  the  number  of  communication  nodes  is  to 
reduce  or  combine  organizations  in  the  external  system.  The  method  by  which  this 
could  occur  is  well  beyond  the  scope  of  this  research,  but  the  effect  on  the  operator 
would  be  a  reduction  in  the  number  of  organizations  with  which  they  would  have  to 
communicate,  presuming  that  the  newly  combined  organizations  would  communicate 
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more  effectively  within  themselves. 

In  this  system,  the  navigation  task  represents  what  would  be  secondary  tasks  in 
other  systems.  Offloading  secondary  tasks  during  periods  of  high  mental  demand  can 
be  accomplished  with  automation,  which  is  already  being  instituted  in  the  military 
example  modelled  for  this  research.  A  software  tool  called  “Internet  Relay  Chat  Co¬ 
ordinate  Extractor  (ICE)”  is  being  implemented  in  a  few  operational  units,  according 
to  SMEs.  The  automation  could  be  tied  to  the  cognitive  arousal  of  the  operator, 
however  and  switched  off  during  periods  of  low  demand  to  allay  boredom.  Secondary 
tasks  could  also  be  offloaded  to  a  contingency  operator,  as  manpower  availability 
dictates. 

3.6.2  Future  Work. 

Future  work  should  define  the  turn-taking  system  of  conversation  in  the  military 
environment.  A  well  defined  turn-taking  system  would  give  a  more  pedigreed  founda¬ 
tion  upon  which  future  communications  models  could  be  built.  Investigations  should 
also  be  conducted  on  the  effects  of  changing  the  external  system  on  the  operator  at 
an  enterprise  level. 
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IV.  Conclusion 


4.1  Discussion 

Implications  for  future  systems.  In  the  early  stages  of  development,  sys¬ 
tem  architects  have  the  powerful  opportunity  to  determine  how  a  new  system  will 
integrate  with  existing  systems  to  accomplish  a  desired  capability.  If  communication 
and  team  structure  is  not  considered  in  this  integration  process,  new  systems  which 
require  real  time  distributed  team  communications  will  undoubtedly  place  higher  de¬ 
mands  on  operators  as  missions  become  more  complex  and  rely  more  on  collaboration 
for  their  success.  From  a  system  of  systems  standpoint,  these  considerations  were  not 
made  for  the  MQ-1/9,  leading  to  immense  variance  in  task  loading  during  missions. 
Interviews  showed  that  operators  may  swing  from  being  bored  to  overwhelmed  in 
a  matter  of  minutes.  Future  system  designs  need  to  incorporate  an  understanding 
of  the  effective  network  size  from  the  perspective  of  each  operator  in  the  system  to 
optimize  information  flows.  This  research  shows  how  this  understanding  can  be  in¬ 
corporated  into  the  design  through  modality  allocation  and  team  structure,  which 
can  each  be  dictated  in  early  stages  of  development  when  design  changes  are  the  least 
costly.  Multiple  team  structures  should  be  considered  and  potentially  simulated  for 
various  tasks  in  early  development.  Team  structure  could  be  designed  to  vary  during 
the  mission  to  accommodate  dynamic  shifts  in  workload. 

Implications  for  future  analyses.  Previous  works  involving  IMPRINT 

have  not,  to  the  author’s  knowledge,  included  a  robust  statistical  analysis  of  workload 
data  to  compare  multiple  system  configurations.  This  research  uses  vetted  simulation 
study  methods  such  as  those  outlined  in  Banks  et  al.  (2010);  Mitchell  and  Samms 
(2007)  and  Law  (2006)  to  design  experiments  and  determine  the  appropriate  number 
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of  replications  needed  to  account  for  random  variance.  Multiple  replications  made 
possible  the  application  of  powerful  statistical  methods  to  determine  quantifiable 
results  in  lieu  of  heuristic  recommendations  which  are  more  widely  used  in  workload 
prediction. 

4.2  Future  Work 

MacMillan  et  al.  (2004)  were  able  to  combine  theoretical  framework  with  empir¬ 
ical  research  to  make  conclusive  recommendations  about  team  structure.  This  work 
certainly  contributes  to  a  solid  theoretical  underpinning;  future  studies  should  in¬ 
corporate  human  subjects  experimentation  and  empirical  study  to  test  the  results  of 
this  work.  Multi-modal  communication  systems  such  as  the  one  developed  by  the 
Air  Force  Research  Lab  (Finomore  et  al.,  2009)  could  be  used  in  conjunction  with 
well  developed  RPA  mission  simulators  to  experimentally  test  the  effectiveness  of  dy¬ 
namic  allocation  of  communication  modalities  to  alleviate  communication  workload. 
Discrete  event  simulation  is  a  good  fit  for  modeling  communication  at  the  level  pre¬ 
sented  in  this  research,  though  future  analysts  may  consider  agent  based  models  to 
allow  more  granular  and  semantically  oriented  simulation.  The  results  of  the  agent 
based  model  could  then  be  worked  onto  the  workload  predictive  capabilities  of  IM¬ 
PRINT  to  make  accurate  inferences  about  communication  priorities  and  semantic 
content.  Should  the  opportunity  arise,  future  researchers  in  this  held  should  develop 
a  large  corpus  of  RPA  operator  communications.  Having  a  corpus  of  both  verbal  and 
chat  communications  from  an  actual  mission  would  help  researchers  understand  the 
exact  nature  of  communications  in  this  arena  in  ways  that  interviews  cannot.  This 
corpus  could  be  analyzed  with  latent  semantic  analyses  (Dumais,  2004)  to  obtain 
measures  such  as  the  anticipation  ratio  and  situation  awareness  measures  discussed 
in  MacMillan  et  al.  (2004)  to  assess  the  effectiveness  of  team  communication  in  real 
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missions.  Analyzing  communication  data  from  real  missions  could  be  extremely  diffi¬ 
cult,  so  the  author  suggests  attempts  to  study  a  representative  RPA  related  exercise 
or  war  game  to  gather  data.  As  it  would  be  both  unsafe  and  contemptible  to  perform 
experiments  during  real  missions,  simulators  could  be  used  to  further  validate  team 
structure  and  system  node  changes  and  their  effects  on  workload. 

Remotely  piloted  systems  have  more  communication  channel  options  than  systems 
where  the  pilot  is  in  the  aircraft,  and  this  research  implies  that  more  diverse  communi¬ 
cation  is  not  necessarily  ideal  in  all  conditions.  Many  operators  prefer  mIRC  because 
of  its  effectiveness  and  persistence,  but  it  is  not  a  particularly  rich  media  Robert  and 
Dennis  (2005).  More  connectedness  between  organizations  certainly  enable  coordi¬ 
nations,  but  also  increases  communication  overhead.  Research  to  compare  manned 
and  unmanned  systems  and  their  respective  team  structures  and  communications 
architectures  would  be  valuable. 

IMPRINT  allows  the  analyst  the  ability  to  set  a  workload  threshold  and  simulate 
how  an  operator  might  manage  the  tasks  to  keep  from  being  overwhelmed.  This 
feature  is  particularly  useful  for  determining  which  tasks  are  being  delayed  or  dropped 
at  certain  times  in  the  mission,  and  can  be  used  to  measure  the  number  of  delays  for 
tasks  of  interest.  This  research  shows  that  IMPRINT  to  create  and  measure  multiple 
simultaneous  metrics  to  show  how  changing  the  system  affects  different  aspects  of 
the  operator’s  experience.  The  results  show  that  workload,  monitoring  time  and 
the  number  of  delayed  tasks  are  affected,  but  they  do  not  provide  a  useful  basis  of 
comparison.  Workload  can  be  reduced  by  changing  the  number  of  communication 
nodes,  but  how  much  can  it  be  reduced  before  boredom  or  change  blindness  begin  to 
occur?  Further  work  must  be  done  to  develop  a  theoretical  optimum  between  task 
saturation,  or  ’red- line’  (Grier  et  ah,  2008)  and  boredom,  or  the  ’blue-line’  (De  Waard, 
1996). 
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Appendix  A.  MIC  Questionnaire 


General  Questions 

1.  What  are  your  major  goals  and  functions  as  a  MIC? 

2.  How  long  is  a  typical  mission? 

3.  What  are  the  major  segments  of  your  day? 

4.  Which  channels  do  MICs  use  to  communicate  with  parties  within  the  GCS? 
How  do  they  decide  which  channels  to  use  at  what  times? 

5.  What  is  the  process  used  to  deal  with  large  volumes  of  chat  communication 
events?  Are  there  scan  patterns?  What  are  the  things  you  specifically  look  for? 

6.  How  much  chat  information  simply  needs  to  be  passed  to  another  operator? 
How  much  is  addressed  particularly  to  the  MIC?  How  much  of  it  requires  critical 
thinking,  extra  look-up  tasks  or  problem  solving? 

7.  Which  parties  have  the  highest  priority  during  each  mission  segment? 

Model  Parameter  Specifics 

1.  What  are  the  major  chat  rooms  for  each  segment  of  the  mission? 

2.  How  many  parties  are  in  each  of  the  rooms? 

3.  In  which  rooms  do  you  spend  the  most  time  reading  and  typing? 

4.  Which  mission  segments  are  the  most  heavy  on  chat?  Are  you  able  to  read 
everything  during  that  phase,  or  are  you  skimming/scanning  the  windows? 

5.  How  long  does  it  take  to  scan  the  chat  windows  unencumbered  (in  seconds)? 
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6.  During  each  mission  phase,  how  often  (in  seconds)  do  chat  events  come  in  from 
each  of  the  major  rooms? 

7.  ff  scanning,  how  often  do  yon  stop  and  read  for  comprehension? 

8.  On  voice  channels,  how  often  do  interruptions  occur  with  each?  To  what  are 
the  interruptions  attributed?  How  are  the  interruptions  dealt  with? 

9.  How  often  do  chat  windows  fill  up  and  result  in  excess  scrolling  for  previous 
information?  Is  information  sometimes  missed  completely? 

10.  When  will  you  take  information  given  on  one  channel  and  pass  it  through  a 
different  channel  to  another  party?  How  often  does  this  occur  during  a  typical 
mission? 

11.  As  a  percentage  of  total  chat,  how  much  causes  you  to  type? 

12.  As  a  percentage  of  total  chat,  how  much  causes  you  to  speak  on  the  radio? 

13.  As  a  percentage  of  total  chat,  how  much  causes  you  to  speak  on  the  intercom? 
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14.  ABSTRACT  As  the  United  States  Department  of  Defense  continues  to  increase  the  number  of  Remotely  Piloted 

Aircraft  (RPA)  operations  overseas,  improved  Human  Systems  Integration  becomes  increasingly  important.  RPA  systems 
rely  heavily  on  distributed  team  communications  determined  by  systems  architecture.  Two  studies  examine  the  effects  of 
systems  architecture  on  operator  workload  of  US  Air  Force  MQ-1/9  operators.  The  first  study  ascertains  the  effects  of 
communication  modality  changes  on  mental  workload  using  the  Improved  Research  Integration  Pro  (IMPRINT)  software 
tool  to  estimate  pilot  workload.  Allocation  of  communication  between  modalities  minimizes  workload.  The  second  study 
uses  IMPRINT  to  model  Mission  Intelligence  Controllers  (MICs)  and  the  effect  of  the  system  architecture  upon  them. 

Four  system  configurations  were  simulated  for  four  mission  activity  levels.  Mental  workload,  monitoring  time  and  the 
number  of  delayed  tasks  were  estimated  to  determine  the  effect  of  changing  system  architecture  parameters.  Literature 
and  MIC  interviews  provided  parameters  for  the  model.  The  analysis  demonstrates  that  the  proposed  changes  have 
significant  effects  which,  in  some  conditions,  bring  the  overall  workload  function  toward  a  proposed  theoretical  optimum. 
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